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Abstract 

In this paper, we discuss the positivity of the Hermitian form (, )^ introduced by Jian- 
Shu Li in Let (Gi,G2) be a type I dual pair with Gi the smaller group. Let tt be 

an irreducible unitary representation in the semistable range of 9{MGi, MG2) (see |^). 
We prove that the invariant Hermitian form (, is positive semidefinite under certain re- 
strictions on the size of G2 and a mild growth condition on the matrix coefficients of tt. 
Therefore, if {,)t: does not vanish, 9{MGi, MG2){tt) is unitary. 

Theta correspondence over R was established by Howe in ( [Q). Li showed that theta 
correspondence preserves unitarity for dual pairs in stable range. Our results generalize the 
results of Li for type I classical groups ( [0). The main result in this paper can be used to 
construct irreducible unitary representations of classical groups of type I. 

1 Introduction 

Let (Gi, G2) be an irreducible reductive dual pair of type I in Sp (see Q (l^)- The dual 
pairs in this paper will be considered as ordered. For example, the pair {0{p, q), Sp2n{^)) 
is considered different from the pair (S'p2n(R), 0{p, q)). We will in general assume that the 
size of Gi(Vi) is less or equal to the size of G2(V2). In other words, dim£)(Vi) < dimD(y2)- 
Let Mp be the unique double covering of Sp. Let {l,e} be the preimage of the identity 
element in Sp. For a subgroup H of Sp, let MH be the preimage of H under the double 
covering. Whenever we use the notation MH, H is considered as a subgroup of certain 
Sp. Let Lo{MGi, MG2) be a Schrodinger model of the oscillator representation of Mp. The 
Harish- Chandra module of u!{MGi, MG2) consists of polynomials multiplied by the Gaus- 
sian function. 

Since the pair (Gi,G2) is ordered, we use 9{MGi, MG2) to denote the theta correspon- 
dence from n{MGi,uj{MGi,MG2)) to n{MG2,u;{MGi, MG2) (see §). In this paper, 
whenever we talk about " K -finite matrix coefficients" or "X- finite vectors" of a represen- 
tation TT of a real reductive group G, "K" is used as a generic term for a specified maximal 
compact subgroup of G. Throughout this paper, we will mainly work within the category of 
Harish- Chandra modules. A representation of a real reductive group refers to an admissible 
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representation unless stated otherwise. Throughout this paper, a vector in an admissible 
representation tt means that v is in the Harish-Chandra module of tt which shall be evident 
within the context. 



Let F be a vector space of finite dimension. Let be a subspace of V. A direct complement 
of in F is a subspace U such that 

Now suppose V is equipped with a nondegenerate sesquilinear form (,). The orthogonal 
complement of in consists of 

{v eV \{v,w) ^0 V weW}. 

It is denoted by W-^. 

Let TT be an irreducible admissible representation of AIGi such that 7r(e) = — 1. tt is 
said to be in the semistable range of d(MGi, MG2) if the function 

(c^(MGi,MG2)(5i)0,^)K^(5i)t') (V 0,^ e c^(MGi, A/G2); V i; e tt) 

is in L^~^ (MGi) for all sufficiently small nonnegative S (i.e., 6 £ [0, c] for some c > 0). We 
denote the semistable range by TZs{MGi,uj{MGi, MG2))- Suppose from now on that tt is 
in 'Rs{MGi,Lo{MGi,MG2)). For each e uj{MGi,MG2) and u,w e tt, wc define an 
averaging integral 

/ {lo [MGi , MG2 ) (gi )0, 7/;) (u, ^(51 )v)dg^ 

JMGi 

and denote it by {(j)(E)v, iIj®u)t^. Thus (, becomes a real bilinear form on lo{MGi, MG2)®7r. 
Our definition of (,)t; differs slightly from the original definition of Li in Let 92 G 

MG2 act on Lj (MGi,MG2) ® TT by w(MGi,MG2)(g2) Id. In g, we show that if 
(, )7r 7^ then (, descends into a sesquilinear form on the ii'-finite dual representation of 
9{MGi, MG2){tt). For tt unitary, (, is an invariant Hermitian form on 6{MGi, MG2)(7r). 

For TT unitary, a conjecture of Li says that (, )^ will always be positive semidefinite (see 
lITll). If Li's conjecture holds and (,)^ ^ 0, then 6{MGi, MG2){tt) is unitary. In this pa- 
per, we will prove that (, )-„ is positive semidefinite under certain restrictions. This partly 
confirms the conjecture of Li. The nonvanishing of certain (, )^ is proved in Q| and in 

We adopt the notations from g, Let (Gi(T4), G2(V2)) be a dual pair of 

type I. Suppose V2 = V2 © V2 such that 

1. (,)2 restricted onto is nondegenerate; 

2. Vi = (^2°)^; 

3. V2 is a direct sum of two isotropic subspaces: 

Obviously, V2 will always be of even dimension. Let X''^ = Horn d{Vi,X2)- The oscillator 
representation uj{MGi{Vi), MG2{V^)) can be modeled on L'^{X°). The action of MGi 
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on L'^{X'~') is equivalent to the regular action of Gi on L^(X°) tensoring with a unitary 
character ^ of MGi. The generic actions of Gi on X° are classified abstractly in Theorem 



4.1 and Theorem 4.2 



Later in this paper, the oscillator representation uj{MGi(Vi), MG2(V2)) is denoted as 
uj{M°Gi,M°G^) to indicate the fact that MGi(Vi) in (MGi(Vi), MG(V2)) might be dif- 
ferent from MGi(Vi) in (MGi(Vi), MG2(V2')). For the same reason, the oscillator repre- 
sentation uj{MGi{Vi),MG2ivi)) is denoted by w(M'Gi, M'G^). 

Theorem 1.1 (Main Theorem) Let {Gi,G2) be a dual pair. Let'E.{g) he Harish- Chandra's 
basic spherical Junction of Gi . Suppose tt is an irreducible unitary representation of MGi 
such that 7r(e) = — 1. Suppose 

1. for any x,y G Gi, the functio n S{ xgy) is integrable on Gi^ for every generic cf) G 
HomjjiVijX!^) (see Definition ^.1); 



2. the tensor product t:{) — a;(M'Gi, Af'Gj) ®7r(g)^, considered as a representation of Gi, 
is weakly contained in i^(Gi) (see JT^J. 

Then (, )^ is positive semidefinite. If {, does not vanish, then 9{MGi, MG2){t^) is unitary. 

Remarks: 

1. uj{M'Gi, M'G'2)), TT and ^ are all projective representations of Gi. The fact that ttq 
becomes a unitary representation of Gi is explained in Part II. 

2. The first condition roughly requires that 



A- ^ vo^ ^ dim£,(yi; 



The precise statement depends on the groups involved. The function S(f/)|G'i^ is in 
L^(Gi^) implies that 'E.{xgy)\Gi^ is in L^{Gi^) for any x,y € Gi and vice versa. In 
fact, S((7) is bounded by a multiple of 'E.{xgy) and vice versa. Furthermore, for any 
compact subset F of Gi, there exists a constant G, such that for any x,y G Y, 

E{xgy) < CEig) [g^G,). (1) 

One can prove this by studying the compact picture of the basic spherical principle 
series representation (see Chapter VII. 1 Q). Since this remark may have already 
been in the literature and a proof will incur a new set of notations, we choose not to 
give the proof. 

The growth of matrix coefficients of Lo{MGi{yi), MG2{V2)) can be determined easily. 
Thus the second condition can be converted into a growth condition on the matrix 



coefficients of tt (see Corollary 5.1) 



4. The condition 1 and 2 imply that tt is in 7^s(MGl, cj(MGi, AfG2)). Therefore, (, )^ is 
an invariant Hermitian form on 9{MGi, MG2){tt). The unitarity of 9{AIGi, AfG2)(7r) 
follows since (, is positive semidefinite. 

This paper is organized as follows. In Part I, we prove some positivity theorems in the sense 
of Godement ||^ . In Part II, we construct the dual pair (Gi , G2) in terms of homomorphisms 
and study various subgroups and liftings concerning the tensor decomposition 

u{MGi,MG2) ^ uj{M°Gi, M^Gl) ® uj{M' Gi, M' G'2). 
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This tensor decomposition is termed as the mixed model in ||Tl|. The interpretation of 
this tensor product is not completely trivial since MGi, M^Gi and M'Gi may be different 
double coverings of Gi. In Part II, we essentially redo part of section 4 in just to be 
safe. In Part III, we study (w(M°Gi, M^G^), L'^{X°)) and classify aU the generic Gi-orbits 
in X'^ . This enables us to reduce our averaging integral ((/> O u, u).,^ to an integral on 
Gi-orbits: 



/OeGi\X0 JGi Jx£C 

We study each generic orbit integral 



(f>{g x)(j){x){u,Tro{g)u)dxdgd[0]. 



(f>{g ^x)(f)ix){u,TrQ{g)u)dxdg 

Gi Jxeo 

in full generality and convert it into an integral on the isotropic group Gi^ 

{T^Q{9)uQ,uo)dg. 



Next, we apply the positivity theorem (Theorem 2.3) to show that this integral is nonneg- 
ative. Thus (, )^ is positive semidefinite. Finally, we take the pair {0(j),q), Sp2nO^)) as an 
example and state our main theorem in terms of leading exponents of tt. 



The author wishes to thank Professors Sigurdur Helgason, Roger Howe, Jian-Shu Li, Tomasz 
Przebinda, Irving Segal, David Vogan Jr. and Nolan Wallach for encouragements, sugges- 
tions and helpful discussions. He also likes to thank Marie Hutchinson for helping him read 
through the paper of Godement ( §). 



2 Part I: Positivity Theorems 

Let G be a real reductive Lie group. Let K he a, maximal compact subgroup of G. For any 
unitary representation (tt, H) of G and any a ^ K, let Her be the X-isotypic subspace of 
H. Let S be a subset of K. We denote 

by H{S). 



2.1 A generic Theorem 

Theorem 2.1 Let G be a real reductive Lie group. Let K be a maximal compact subgroup 
of G. Let S(.g) be Harish- Chandra's basic spherical Junction with respect to K. Let H 
be a closed unimodular Lie subgroup of G. Suppose that S((7)|jy is in L^{H). Let (j) be a 
positive definite function in L'^~^'^(G){S) for some finite subset S of K and any e > 0. Then 
J^(j){h)dh > 0. 

Here L'^^^{G){S) is defined with respect to the left regular action of G. 

Proof: By the GNS construction, we construct a unitary representation (a, Ti.) such that 
4'i9) — (o'(.9)'7! v) for some cyclic vector 77 in 7i{S). Since (j) is a positive definite function in 
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L^+'^(G) for any e > 0, by Theorem 1 in u is weakly contained in L'^{G). Thus, there 
exists a sequence of convex hnear combinations of diagonal matrix coefficients of L'^{G){S) 

Mg) = ^a«(i(5)«f\.«), ^af) = 1 K« G i^(G)(5),af) > 0) 
1=1 1=1 

such that 

Mg) (t>{9) 

uniformly on compacta. Let Cc{G){S) be the space of continuous and compactly sup- 
ported functions in L^{G){S). Since CciG){S) is dense in L^{G){S), we choose uf^ to be 
in Cc (G)(5). Notice that 

h 

A,(e)=^af)||uf)||i. -></)(e) = ||r;f 
1=1 

Hence {Ai{e)}°Zi is a bounded set. Suppose Ai(e) < G. From Theorem 2 in 

|(L(5)uf\z.«)|<hf)||i.(5]d(a))^S(g). 

o-es 

It follows that 

u 

\A{g)\=\J2af\Lig)uf\uf^)\ 
1=1 

<G(Erf(^))^S(5). 

We have proved that (f>{g) can be approximated by positive definite functions Ai[g) such 
that Ai(g) arc uniformly bounded by a fixed multiple of 2(17). 

Now consider the restrictions of (f){g) to H. From (22.2.3) in ||], for {L{g)ui\u:P) with 
Uj-''* a compactly supported continuous function, 

/ {L(h)i^,t^)dh>(). 
Jh 

Thus Ai{h)dh > 0. But Ai{g)\H are bounded by a fixed multiple of an integrable function 
S((7)|/f . By the Dominated Convergence Theorem, 

/ (j){h)dh = lim / A,{h)dh > 0. 

Q.E.D. 
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2.2 First Variation 

Theorem 2.2 Let G be a real reductive Lie group. Let K be a maximal compact subgroup 
of G. Let H be a closed unimodular Lie subgroup of G. Let 'B.[g) be the basic spherical 
function of G of Harish- Chandra. Suppose that 'E.[g)\H is in L^{H). Suppose (7r,7i) is an 
irreducible unitary representation weakly contained in L^{G) (see Let 



k 

V 

i=l ' 



/ Mx)'^i7iix))udx 
Jm 



where 

• u is a K-finite vector in Ti.; 

• M is a smooth manifold; 

• (f>i is continuous and is supported on a compact set Xi C M ; 

• ji : AI ^ G is smooth except a codimension 1 subset and the closure of 7i(Xi) is 
compact. 

Then 

/ {TT{h)v,v)dh > 0. 
Jh 

The basic idea is to control the function {TT{g)v, v) by a convergent integral of left and right 
translations of S(g). 



Proof: From the proof of Theorem 2.1, we have a sequence of if-finite compactly sup- 
ported continuous positive definite functions 

An{g) {Tr{g)u,u) 

uniformly on any compact subset and 

\A„,ig)\<CE{g). 

This implies that 

\Am{xgy)\ < CE{xgy). 
By the compactness of supp[(j)i) and the unitarity of tt, 

{'K{g)v,v)=^ / (l)i{x)(t>j{y){TT{g-^,{x))u,Ti{-fj{y))u)dxdy. 

Since the closure of ji{Xi) is compact, the closure of jj{Xj)^^ gji{Xi) is compact for every 
g ^ G. By the Ineqaulity |l], for any to, 

I / Mx)(f>j{y)Amhjiyy^gjiix)))dxdy\ 



k 



MxM 



i.J=l 

<CiEig) 



<gY, / \MxmAy)\^iijiyr'9iri^)))dxdy 

, „_i J MxM 
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for some Ci > 0. Furthermore, 

k 



V / (l}i{x)(l)j{y)A^{-fj{y) ^g-fi{x))dxdy ^ {Tr{g)v,v) 
pointwisely as m ^ cxd. By the Dominated Convergence Thereom, 

k 

/ {■K{h)v,v)dh = hm / / (l)i{x)(t)j{y)Am{l]{y)~^h'^i{x)))dxdydh. 

Jh Jh JmxM 

''ij — -L 



But 

1=1 

For each 



/ (t>r{x)(t)j{y){L{-ij{y) ^h-f,{x))u^^ ,u^^)dxdydh 

k p k p 



H ,_i JM ,_i JM 



(4) 



>0 



because Im 4'iix)Li'Jiix))um dx is a continuous and compactly supported function on 



IM 

G. Hence for every m 



/ (j}^{x)(j)j{y)Am{lj{y) ^hji{x))dxdydh>0. 



H j J M xM 



It follows that 

{Tr{h)v,v)dh > 0. 



H 



Q.E.D. 



2.3 Second Variation 



Theorem 2.3 Let G be a real reductive Lie group. Let K be a maximal compact subgroup 
of G. Let H he a closed unimodular Lie subgroup of G. Let 'E.{g) be the basic spherical 
function of G of Harish- Chandra. Suppose that S(g)|if is in L^(H). Suppose (7r,7i) is an 
irreducible unitary representation weakly contained in L^{G) (see JT^J. Let 



i=l 

where 

• Ui are K-finite vectors in 7i; 



n „ 

v = y2 (l)i{x)n{'ji{x))uidx 

■1 JM 
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• M is a smooth manifold; 

• (pi is continuous and is supported on a compact subset Xi C M ; 

• : M G is smooth except a codimension 1 subset and the closure of ^i{Xi) is 



compact. 

Then 

{TT{h)v,v)dh > 



H 

IS 



The only difference from Theorem 2.2 

n ^ 

v = y2 4'i{x)TT{ji{x))Uidx 



i=l 

instead of 



V = 

4=1 ■ 

Proof: Let V be the hnear span of 



II, „ 

/ (j)i{x)TT{'yi{x))udx. 
.-1 Jm 



{TT{k)ui I i G [l,n],k G K}. 

Since Ui are K-fmite, is a finite dimensional representation of K. Let u be a iiT-cyclic 
vector in V. Let C{K) be the space of continuous functions on K. Consider the action of 
C{K) on u: 



Tr{f)u^ / f{k)Tr{k)udk. 
Jk 

Apparently, Tr{C{K))u = V. Let 

fi{k)Tr{k)udk. 



K 



Then 



n ^ 

w^V" / 4>i{x)T:{-ii{x))uidx 

= / 't'ii^)'^hii^)) / fi{k)'n-{k)udkdx (5) 
i=i Jk 

= ^ / / (t)t{x)fi{k)TT{'-fi{x)k)udxdk 



i=l 



M JK 



Apply Theorem 2.2 to functions 4'i{x)fi{k) on M x K and 

7* : {x,k) <^ M X K ^ "/r{x)k e G. 
The conclusion follows immediately. Q.E.D. 

Conjecture 1 Let G be a real reductive group. Let K be a maximal compact subgroup of 
G. Let 'E,{g) be Harish- Chandra's basic spherical function. Let H be a subgroup of G such 
that ^{g)\H is in L^{H). Let (f>{g) be a positive definite continuous function bounded by 
E{g). Then ^^4>{h)dh > 0. 
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3 Part II: Dual Pairs and Mixed Model 



The basic theory on the mixed model of the oscillator representation is covered in p| with 
reference to an unpublished note of Howe. We redo part of section 4 of |^ with emphasis 
on the actions of various coverings of Gi regarding the mixed model 

uj{MGi,MG2) = uj{M°Gi,M°G°2) ® uj{M'Gi, M'G'^). 

Let Vi be a vector space over D equipped with a sesquilinear form (,)i, V2 be a vector 
space over D equipped with a sesquilinear form (,)2- Suppose one sesquilinear form is [t- 
Hermitian and the other is (j-skew Hermitian. Let Gi be the isometry group of Let 
V = HomniVi, V2) be the space of Z3-linear homomorphisms from Vi to ¥2- 

3.1 Setup 

Let 0, i/" G V^, vi,ui e Vi and V2 G V2. We define a unique 0*(w2) such that 

{<t>*{v2),Vi)i = {V2,(l){vi))2. 

It is easy to verify that (jf € Homo{V2,Vi). Thus we obtain a * operation from V to 
V* = HomD{V2,Vi). Let a e M. Then 

((a0)*(t)2),wi)i = {v2,acj){vi))2 = a{v2,(j){vi))2 = a((/i* (^2), = (a^*(w2), 
Therefore, the ^-operation is real linear. 

Let tr{*) be the real trace of a real linear endomorphism. Since V and V* are real vector 
spaces, we can now define a real bilinear form Vl on V as follows 

VL{cj>,^)=tr{rcl>). 

We observe that 

{r<I^M,v[), = {<j>{v^),^{v[))2 = ±mv'l)A{vi)f2 = ±{<t>*Wl),Vl){ - 

Define a *-operation on Endo{Vi) by 

Then, (0*^/;)* = It follows that 

0) = tr(0» = <r(((/)*i/')*) = tr(-i/'*'/') = -^^(0, -0)- 
It is easy to verify that is nondegenerate. Therefore, O is a real symplectic form on V . 

Next we define the action of Gi on V as follows 

(.9i4>)(.vi) = ^(sr^i'i)- 

We observe that 

=((3i(^)(wi),(giV')(fi))2 

= {(j){g^^ui),t^{g^\i))2 (6) 

=(V'*'?!'(.9r^"i)>5'r^^i)i 
=(5i('/'*0)5r^"i>^^i)i- 
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It follows that 

^{9i<f>,9i^) = tr{ig,^ng,<j))) = tr{g,r4>gi') = tr{r4>) = ^M). 
Therefore G\ is in Spiy, O). We define the action of G2 on V similarly by 

{924>){vi) = g2(t>{vi). 

One can verify that G2 also preserves i7. In addition, the action of Gi commutes with the 
action of G2. 

3.2 Subgroups 

Let V2 be a D-linear subspacc of V2 such that 

• (, )2 restricted to V2 is nondcgcncrate; 

• There exist isotropic subspaces X2 and Y2 such that 

^2° © y2 = 

Let V2 be the space of vectors perpendicular to ¥§ with respect to (, )2. Write 
X° = HcrniD {Vi ,X°), Y° = Homo {Vi ,Y^), 
V = HomD{Vi,V^), y° = HomD{Vi,V^). 

For any 0, ^ G 

(V'*<^t;i,wi)i = {<t>Vi,tpUi)2 = {vi,ui G Vl). 

Thus f7(V',ct>) = tr{c[)*t/j) = 0. is an isotropic subspacc of {V,fl). For the same reason, 
Y'^ is also an isotropic subspacc of (V, fl). Furthermore, wc have 

V = V'® V°, V° = X°® Y°. 

Let G2 be the subgroup of G2 such that G" restricted to V2 is trivial. Then G2 is isomorphic 
to G2(V2'). Let G'2 be the subgroup of G2 such that Gg restricted to V2 is trivial. Then G2 
is isomorphic to G2(V2)- 

Let be the restriction of fl on V°. Let fl' be the restriction of ^ on . Then Sp{V", Q") 
and , fi') can be embedded into Sp{V, Q) diagonally. Let GL{X^, Y^) be the subgroup 
of Sp{V°,ilP) stabilizing X^ and Y^. Since Gi and G2 act on V^, we obtain a dual pair 

(Gi,GO)C5p(yO,QO). 

We denote this embedding by On the other hand, since Gi and G2 act on V, we obtain 
another dual pair 

{Gi,G'2)cSp{V',n'). 

We denote this embedding by i'. Now the group Gi is embedded into Sp{V,^) by i° x i'. 
We denote this embedding by i. 
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3.3 Metaplectic Covering and Compatibility 

For any symplectic group Sp, there is a unique double covering MSp. We call this the 
metaplectic covering. Let e be the nonidentity element in MSp whose image is the identity 
element in Sp. For any subgroup G of Sp, let MG be the preimage of G under the meta- 
plectic covering. Then every MG contains e. 

Let M^Spiy^, n"), M'Sp{V', fl') and MSp{V, fl) be the metaplectic coverings of S'p(yO, 17°), 
Sp{V', n') and Sp{V, f2) respectively. Let M°, M' and M be the covering maps respectively. 
When we consider Sp{V'^, Cl'^) as a subgroup of Sp{V, O), we obtain a group MS'p(y°, Cf). 
On the other hand, Sp{V^,Q,^) has its own metaplectic covering, namely, M^Sp{V^,fl^). 

Lemma 3.1 (compatibility) The group MSp{V°,fl°) is isomorphic to M°Sp{V°,Q°). 

Proof: It suffices to show that MSp{V°,n^) does not split. Suppose MSp{V°,n°) splits. 
Let be a maximal compact subgroup of Sp(V, fl) such that = K C] SpiV'^, il") is a 
maximal compact subgroup of Sp{V^ Then MK'^ splits. On the other hand, K can 
be identified with a unitary group U. The metaplectic covering of U can be represented by 

{{t9)\e =detg,geU} 

For the subgroup K^, we see that MK^ must be the nontrivial double covering of K^. It 
does not split. We reach a contradiction. Q.E.D. 

This Lemma basically asserts that if a smaller symplectic group is embedded in a big- 
ger symplectic group canonically, then the metaplectic covering on the smaller group is 
compatible with the metaplectic covering on the bigger group. Let 

iP : (M°Gi,MOG^) C M^Sp{V'^,n^) 

be the lifting of i^. Let 

i' : {M'GuM'G'^) C M'Sp{V',n') 
be the metaplectic lifting of i'. Let 

i : {MGi,MG2) C MSp{V,n) 

be the lifting of i. According to the compatibility lemma, we may consider M^Sp{V°,Q^) 
and M'Sp{V' as subgroups of MSp{V,Cl). These two subgroups intersect. The inter- 
section is {1, e}. 

Consider the natural multiplication map 

j : M^Sp{V^,il^) X M'Sp{V',fl') MSp{V,Q). 
Its kernel is {(1, 1), (e, e)}. If 5 e Gi, then 

i{g) = (z°(.g),x'(.g)) e x Sp{V',n') c Sp{V,n) 

The covering group MGi is then isomorphic to the quotient 

{Jig^g') I 5° e M^Gr,g' e M'Gr,M\g') = g = M'{g')}/{{1, 1), (e, e)}. 
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Lemma 3.2 Each element in MGi can he expressed as i{g^,g') with 
up to a factor of 

{(1,1), (e,^)}. 

Lemma 3.3 As a group, 

M^'Gi = {{g,g') \ M{g) ^ M' {g'), g £ MGi,g' G Af'Gi}/{(l, 1), (e, e)}. 

3.4 Oscillator Representation as tensor product 
Theorem 3.1 The representation 

uj{M^Gi,M^Gl)®oj{M'Gi,M'G'2) 

restricted to 

b{cf,9') I cf e M"G,,g' e M'Gi,MO(g") - g = M'ig')} 
descends into uj{MGi, MG2)\mGi- 
Proof: Suppose g £ MGi. Then g can be written as 

(5°,<?') I 5° e A^°Gi,5' e A/'Gi,M"(50) = M'{g') 
up to a multiplication by 

{(1,1), 

It is easy to see that 

oj{MGi,MG2){l: 1) = id = cj(Af°Gi, Af°G§)(e) (g) uj{M'Gi,M'G'2){e) 
It follows that 

w(AfGi,AfG2)(5) = t^(A//°Gi, A/°G°)(<?°) ® u;(A//'Gi,A//'G^)(<?') 
Our theorem is proved. Q.E.D. 

Let TT be an irreducible unitary representation of AfGi in the semistable range of ^(A/Gi, AfG2) 
such that 7r(e) = —1. Identify the representation uj{MGi, MG2y ® tt with 

uj{M°Gi,M°G°2y<S>{uj{M'Gi,M'G'2y(E)Tr). 

From Lemma B.3, g^ € A/^Gi can be represented by a pair {g, g') up to a multiplication of 



(e, e). Since 

u;(A//'Gi,A/'G^)'=(e)7r(e) ^ id, 

we can write 

{uu{M'Gi,M'G'2Y®Tr){g^^)^u{M'Gi,M'G'2Y{g')®Tr[g). 



The proof of Theorem shows that 

w(Af'Gi,Af'G'2)^®7r 
can be regarded as a unitary representation of Af^Gi. 
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3.5 Schrodinger Model of cj(M°G'i, M°G9,) 

Recall y° = © Y° and both Y° are Lagrangian in Let GL{X°, be the 

subgroup of Spiy°,n°) stabilizing X° and Y°. Then 

GL(X°,y°) = Gi(X") = GL(y"). 

Let L2(X0) be a Schrodinger model of uj{M"Gi, M°G^) (see g, Q). The group 
M"GL(X",rO) acts on L^{X°) naturally. Since Gi is a subgroup of GL{X°,Y°), an ele- 
ment in the group Af^Gi can be written as 

(€,5) l5eGi,^GC 

such that the operator 

(c.(AfOGi,M"GO)(e,5)0)(^) = ^'/'(s-'^:) {x e e 

is unitary. 
Consider 

/ (c.(A/°Gi,M0G°)(C,5)'^,^)(w, (c^(M'Gi,M'GJ,)^ ® 7r)(^,g)«)dgdC (7) 

JMOGi 

with u, w G a;(Af'Gi, Af'G2) (Xitt. Since the group action of Gi on L'^{X'^) is already unitary, 
is a unitary character of M°Gi. Thus ^ O w(Af"Gi, Af^Gj) can be viewed as a unitary 
representation of Gi. Moreover, 

?L.(A/°Gi,M°G°)(g,e)0(a:) = 0(5"'^:). 

Define 

TTo = C ® (tj(A//'Gi, A//'G^)" ® tt). 

Viewing {lu{M'Gi, M'Gi^Y ^ tt) as a representation of Af°Gi, ttq descends into a unitary 
representation of Gi . 

Tensor products with ^ here do not change the ambient spaces. However, the group ac- 
tions differ by a unitary character. Now, the integral ( |^) becomes a multiple of 



^x)ip{x)dx{u,T:Q{g)v)dg. (8) 
Gi Jxo 



This integral can be expressed as orbital integral 



Gi JoeGi\x'> Jxeo 

In Part III, we will classify the generic Gi-orbits in and study each generic orbital 
integral 



(j){g ^x)tp{x){u,T:o{g)v)dxdg. 

Gi Jx£0 
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4 Part III: Orbital Integrals 



Recall that — _ffoTOD(Vi, Xj). We need to classify the orbital structure of the Gi-action 
on Let m = dim/j Vi and dimu X2 = p- If m < p, (Gi, G2) is said to be in the stable 
range. The action of Gi on is almost free. This case is already treated in For 
(Gi, G2) in the stable range, our approach can be simplified and in deed coincides with Li's 
approach in From now on, assume m > p. The set of nonsurjective homomorphisms 

from Vi to X2 is of measure zero. Hence we will focus on surjective homomorphisms in X'^ . 
We denote the set of surjective homomorphisms by Xq. Let (p G Xq. 

4.1 The Isotropic subgroup Gi^ 

Let ei, 62, . . . , Cm be a D-linear basis for Vi, and /i, /2, • ■ • , /p be a D-linear basis for 
Then (j) is uniquely determined by 

0(ei),0(e2), . . . ,0(em). 

We will determine the "generic" isotropic subgroups of the Gi-action on Xq. Suppose 
g e Gi stabilizes (j). In other words, 

cj^iu) ^ {gq^){u) ^ cl,{g-'u) (V u). 

This implies that ker((/)) is stabilized by g. Therefore ker((/))-'- is also stabilized by g. 

Lemma 4.1 Let g e Gi and 4> G X^. Then 4> is fixed by g if and only if any vector in 
ker(0)^ is fixed by g. 

Proof: Suppose ip is fixed by g. Let (w,ker(/))i = 0. We choose an arbitrary u € Vi. 
Since 4>[g^^u) — (l>{u), g~^u — u G keicf). This implies that {v,g~^u — u)i — 0. Thus 
{gv,u)i = {v,u)i for every u G Vi. It follows that gv = v. g fixes every vector in w G ker^^. 

Conversely, suppose gv ~ v for any (u,ker0) = 0. We choose an arbitrary u € Vi. Then 
{gv — v,u)i = 0. Hence {v,g~^u — u)i = for every v G ker^^. From the nondegeneracy 
of(,)i, 

g ~ u £ (ker (j)-^)^ = kei (j) 
Therefore, 4>{g~^u — u) = for every u G Vi. It follows that g(j) = 4>- Q.E.D. 

Theorem 4.1 Let (f) be a surjective homomorphism from Vi to X2. Then the isotropic 
subgroup Gi^ is the subgroup that fixes all vectors in ker((/))^. 

The restriction of (, )i onto ker0^ contains a null space, namely, 

= ker^nker^^. (9) 

W is an isotropic subspace of Vi and it may or may not be trivial. Let J7 be a direct 
complement of W in ker (/)-'-, i.e., 

U (BW = keT(l3-^. (10) 

Then (, )i restricted to U is nondegenerate. Thus (, )i restricted onto U-^ is a nondegenerate 
sesquilinear form. Since the group Gi^ fixes all vectors in ker^^ and U C ker0^, Gi^ can 
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be identified with the subgroup of Gi{U^ that fixes all vectors in W. 

From Equations p , p^ , ker0 is the orthogonal complement of W in U-^. From Equ. 
(ll), is a 



tions (28), (29) in Gi^ is a twisted product of Gi(ker(/i/VF) with a at most two-step 

nilpotent group N. 

Theorem 4.2 For orthogonal groups, we take Gi = SO{p,q). The isotropic subgroup Gi^ 
is a twisted product of a classical group of the same type with a at most two-step nilpotent 
group N . It is always unimodular. 

Proof: To show that Gi^ is unimodular, one must show that the adjoint action of Gi (ker 4>/W) 
on the Lie algebra n has determinant 1. This is obvious since n as a Gi-module decomposes 
into direct sum of of trivial representations and the standard representations. Q.E.D. 

4.2 Generic Element 

The homomorphism (p induces an isomorphism 

[(/)] : Vi/kercl)^ 

Notice that ker (p can be regarded as a point in the Grassmannian Q(m, m — p). We obtain 
a fibration 

GLp{D) ^ X° ^g{m,m-p). 

The projection maps </> to kert/). The fiber contains all isomorphisms from Vi/keri/) to 
Thus the fiber can be identified with GLp{D). 

Definition 4.1 Generic elements in are those surjective (j) such that 

1. either (,)i restricted on ker(0) is nondegenerate; 

2. or if the above case is not possible, 

dimD(ker((/>) n ker((/))-'-) = 1 

Let Xqq be the subset of generic elements. The subspaces ker((/)) in Q{m,m — p) are called 
generic [m—p)- sub spaces. The set of generic {m^p)- sub spaces is denoted by Qf){m,m~p). 

Consider the following fibration 

GLp{D) Xo"o g^{m,m-p). 

Since the set go{m, m—p) is open and dense in g{m, m — p), the set Xqq is open and dense 
in Xq. Therefore, Xqq is open and dense in X^ . 

First, suppose (,)i restricted to ker((/)) is nondegenerate. We must have 

ker((?!)) ® ker((/))-^ Vi 

The isotropic subgroup Gi^, can be identified with Gi(ker((/>)) by restriction according to 



Theorem 4.1. It is a smaller group of type Gi . The group Gi^ is automatically unimodular. 



Secondly, suppose 

dimi5(ker((/«) n ker(0)^) = 1 

Notice that this case does not occur for 0{p,q). From Theorem 4.2, Gi^ is a unimodular 
group. We obtain 
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Corallary 4.1 For any generic element (p G ^ooj isotropy subgroup Gi^ is always 
unimodular. 

4.3 Averaging Integral Revisited 

Let TT be an irreducible unitary representation in the semistable range of 9{MGi, MG2)- 
Recall that 

TTO = Uj{M'Gl,M'G'2T «) TT (g) C 

is a unitary representation of Gi . Consider the integral 



<f>{9 ^x)^p{x)dx{u,TTo{g)v)dg (11) 

Gi Jxo 

where (j), ip are X-finite vectors in L'^{X'^) and u, u G ttq. 

Theorem 4.3 Let tt be an irreducible unitary representation in the semistable range of 
0{MGi,MG2). Let 0,^' be in t he H arish- Chandra module of uj{M"Gi, M"G^). Let u,v G 
ttq. Then the function (l){g~^x)xp{x){u,TrQ{g)v) is continuous and absolutely integrable on 
Gi X . Therefore we have 



(f>{g ^x)ij{x)dx{u,7rQ{g)v)dg ^ / / ^x)ij{x){u,Troig)v)dgdx 

Gi Jxo Jx" JGi 

From our discussion in Part II, the integral |l^ is a form of the averaging integral under the 
mixed model 

Lu{MGi,MG2) = uj{M°Gi, M^Gl) ® uj{M' Gi, M' G'^). 



The absolute integrabilty of 0(g ^x)%l:{x){u, Tro{g)v) is guaranteed by the semistable condi- 
tion (see 1^). We skip the proof. 

4.4 Orbital integral in General 

First, let me quote a simplified version of Theorem 8.36 from 

Theorem 4.4 Let G be a unimodular group and H be a closed unimodular subgroup of 
G. Let dg and dh be their Haar measures, respectively. Then up to a scalar, there exists a 
unique G-invariant measure d[gH] on G/H. Furthermore, this measure can be normalized 
such that for any L^ function on G, 



f{9)dg^ [ [ f{gh)dhd[gH]. 

J G/H JH 



'G J G/H J H 

Suppose r is a unitary representation of G, u and v are K finite vectors in r. 
Theorem 4.5 Let G be a real reductive group, and M be a G-homogeneous space. 

• Let Xq be a fixed base point and Gq be the isotropy group of xq . Suppose that Gq is 
unimodular. Then M is isomorphic to G/Gq and possesses a G-invariant measure. 

• Let J : M ^ G be a smooth section of the principle bundle 

B : Go ^ G ^ M 
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except for a subset of at least codimension 1. Assume 4'{y) is an absolutely integrable 
function on M . Then 



vo= <f){y)T{i{y) )vdy 
Jm 

is well-defined. 

Assume (j){g^^xo){u,T{g)v) is integrable as a function on G. Then we have 
(j>{gxQ){T{g)u,v)dg = / {T{go)u,VQ)dgo. 

JGo 



Proof: (1) follows directly from Theorem 4.4 by identifying M with G/Go- 
Since t is unitary and is integrable, vq is well-defined. (2) is proved. 
Notice that ■^{y)GoXo — y. We compute 

H9Xo){T{g)u,v)dg 

H99oXo){T{ggo)u, v)dgod[gGo] 



lg]eG/Go JGo 

Hy) / iT{j{y)go)u,v)dgody 
yeM JGo 

Hy) / iTigo)u,T{-f{yy^)v)dgody 

M J Go 



(t(5o)w, / Hy)Thiy) ^)vdy)dgQ 

Go JM 

{T{gQ)u,vo)dgo. 



(12) 



Q.E.D. 

We can further utilize the right invariance of the Haar measure on G by changing xq into 
an arbitrary x G M. 



Theorem 4.6 Under the same assumptions from Theorem |^.4 suppose tp{x) is an abso- 
lutely integrable function on M . Let 



uq = 'iIj{x)t{j{x) ^)udx 
Jm 

Suppose the function 

(t){g^^ x)'ip{x){u,T{g)v) 

is in L^{G x M). Then we have 



(j){g ^x)ip{x){u,T{g)v)dgdx ^ / iT{go)uo,Va)dga 



M JG 
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Proof: First of all, since r is unitary and ip{x) is integrable, uq is well-defined. According 
to Fubini's theorem, we can interchange the order of integrations. We obtain 



(j){g x)tp{x){u,T{g)v)dgdx 



M JG 



(l){gx)ip{x) {T{g)u, v)dgdx 



M JG 



tpix) I <j){gx){T{g)u,v)dgdx 

M 



<l^{9l{x)xo){T{g)u,v)dg)dx 

M JG 



Tp{x){ (j){gxo){T{gj{x) )u,v)dg)dx by the right invariance of dt; 

M JG 



<l^i9Xo){T{g)T{j{x) )u,v)dg)dx 

M JG 



tp{x) / {'T{go)T{'y{x) ^)u,vo)dgodx by Theorem 4.5 

M JGo 



('r(ffo)(/ '4'[x)T{-f{x) ^)udx),vo)dgo 

, JM 

{T{go)uo,vo)dgo. 

JGo 

Q.E.D. 

4.5 Orbital Integral l{(j),u,0^) 

Let Ox be a generic Gi-orbit in Xqq. Then Ox possesses an Gi-invariant measure. Let tt 
be a unitary representation in the semistable range of 6'(MGi, MG'2). Let us recall some 
notations and facts from Part IL 

1. £, is a. central unitary character of M^Gi and any element 17° in il/"Gi can be expressed 
as a pair (^,(7) with g in Gi. 

2. ttq — uj{M'Gi, M'G'2Y ® TT ® ^ is a representation of Gi. 
We fix a finite vector u in tt ® ^. Suppose 

s 

= 
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with e uj{M^Gi,M°Gl) and (t>[ G uj{M'Gi, M'G'2). Then we have 
= / iLo{MGi,MG2)mA)i^,A9)u)dg 

JMGi 
i Jm<>Gi 

= V / ((c.(Af"Gi, Af°G°) ® ?)(5°)</>°, 0°)(</';- ® (a;(Af'Gi, Af'G^)^ ® ^ ® ?)(^?°)(</'^ ® u))(i5° 



^Y. [ I <l>%g~^x)4>%x)dx{(t)'^®u,T:o{g)[c^^®u))dg 



/ </'°(.g"'a:)(/.O(x)^®M,7ro(5)(0-®u))d5da: 

(14) 

First of all, due to Theorem 4.3, the above integral converges absolutely. Since Xqq is open 
and dense in X^, 



2 



^x)(jP^{x){(l)'j®u,T:o{g){(l)[®u))dgdx 



converges absolutetly. Due to Fubini's Theorem, for almost all the orbits Ox in X^q, the 
function 

4P,{g~^x)4,%x){c^'j (g> u, Tio{g){c^[ ® u)) V 

is absolutely integrable on Ox y. Gi. Secondly, since {(j>^}j^i are rapidly decaying functions 
in the Schrodinger model of w(A'/°Gi, Af^Gj)), {f^^lj^i are absolutely integrable on Xqq. 
Hence are absolutely integrable on almost every Gi orbit Ox- 

Take M to be an Gi-orbit Ox such that 

1. (/i^ is absolutely integrable on Ox for every j; 

2. The function 

cj>%'^x)^{x){4>'^®u,M9){4>[®u)) 

is absolutely integrable on Ox x Gi for every i, j G [1, s\. 
Denote the orbital integral 

Hj^ X ^'^^^^'^'^^'W^^'t)'j®u,^o{g){(l)\®u))dgdx 

by I{(j),u, Ox)- Take t to be ttq. Since A/ can be identified with Gi/Gi^,, Gi forms a fiber 
bundle over M. By local triviality, we choose a smooth section 7 : A/ ^ Gi over an open 
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dense subset of M . Then Theorem 4.6 imphes 



5Z / / <l^% ^x)(j3'l{x){(j)]®u,T:o{g){4'',®u))dgdx 

I j Jo, J Gi 

Y.I (Mgo) [ '^)Ml{yr'M®n)dy, [ '^)^a{^{y)-^M®u)dy)dg^ 
TIJg^, Jo, Jo, 

{T^Q{go)uo,uo)dgo. 

Gi, 

(15) 



Here 



O^ 



4.6 Compactly Supported Continuous Functions 

The theorems we have so far proved hold for compactly supported continuous (not necessar- 
ily smooth) functions 4>^,^l^^ as well. In fact, any compactly supported continuous function 
on can be dominated by a multiple of the Gaussian function fi{x) on X'^. Therefore, 
the function 

\<j>^ig-'x)i^)i^'^ ® u, MsM ® ^))l 

is always in L^{Gi x X'-^). The rest of the argument from Part III goes through. Again, we 
obtain 

Theorem 4.7 Let n be a unitary representation in the semistahle range of 6{MGi,MG2)- 
Let u he a K -finite vector in tt ® ^. Let 4>^ he compactly supported continuous functions on 
X° and € w(M'Gi, M'G^). Write 

S 

i=l 

Then the integral 

(^ (g) u, (/) (g) u)^ = 2 ^ / / (l)°{g-^x)(l)°{x){(l)'j (g)U,Tro{g){(f)'i<S>u))dgdx 

is absolutely convergent. For almost every Gi-orhit O (except a subset of measure zero), 
u, O) converges absolutely. Fix such an orhit Ox and a base point x. Choose any smooth 
section 7 : Ox — > Gi over an open dense subset of Ox ■ Let 



"0= L II'^°(2^)'^o(7(y) ')(0-®u)rfy. 

Then 



o. 



I{(l),u,Ox)= {no{g)uo,uo)dg. 

'Gi, 
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5 Part IV: Positivity and Unitarity 

Lemma 5.1 Suppose -k is a unitary representation in TZ{MGi,Uj{MGi, MG2)). Suppose 
for every (f> € lu{MGi, MG2) and a fixed nonzero u ^ tt 

{(j)(E)u,(l)(g)u)^ > 0. 

Then is positive semidefinite. If does not vanish, Then 0{MGi, MG2){'^) is uni- 
tary. 

A similar statement can be found in [p^ . 

Proof: If (, vanishes, the lemma holds automatically. Suppose (, )^ does not vanish. 
Let TZt^ be the radical of (, )^. The linear space 

{V (E)u)/{TZ^n{r (E)u)) 

must be nontrivial. Otherwise V ^ u C TZ^. Since 7?.^ is a (gi, AfiiTij-module, by the 
(gi, Afifi)-action, 

This contradicts the nonvanishing of (, 
Observe that 

(v u) / {n^ n (v u)) 

is an admissible Harish-Chandra module of MG2- From Theorem 7.8 it must be 

irreducible and equivalent ioV ® -k'^ /IZj;. Since 

{<l>,^{9)<l>){'^ig)u,u)dg > 

MGi 

for a fixed u G tt and any _R"-finite 0, (, )Tr\'P(gju induces an invariant positive definite form 
on e{MGi,MG2)in). Thus 6'(MG'i, MG2)(7r) must be unitary. Consequently, (, )^ must 
be positive semidefinite. Q.E.D. 

5.1 Proof of the Main Theorem 

Theorem 5.1 Let R{g) be Harish- Chandra's basic spherical function of Gi. Suppose 

1. TT is a unitary representation in the semistable range of 9(MGi, MG2) ■ 

2. For any x,y ^ Gi, the function 'E.{xgy) is integrable on Gi^ for every generic (j) S 



Hom]:){Vi, X2) (see Definition ^.1) 
3. ttq is weakly contained in L^(G'i). 
Then (, )^ is positive semidefinite. If {, does not vanish, then 6{MGi, MG2){t^) is unitary. 

Roughly speaking, the second condition requires Gi^ be half the "size" of Gi. The first 
condition is redundant assuming the second and the third conditions are true. The third 
conditions can be converted into a growth condition on the matrix coefficients of tt. 
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Proof of the Theorem: Let w be a fixed _Rr-finite vector in tt ® ^. Write 

s 

5 = {0 - ^ ^ I 0° G C^{X'>), cj,[ e u{M'G,,M'G'^)}. 
1=1 

Let (p & S. Choose an arbitrary Gi-orbit Ox in Xqq such that /(</), u, Ox) converges abso- 
lutely. There is a canonical fibre bundle 

Gix Gi Ox- 
Fix a smooth section 7 : Ox Gi over an open dense subset of Ox such that the closure of 
^[supp{4>^)) is compact for every i. Let 

"0= / YA^Ml{y)~^){<l^'^®u)dy. 
From Theorem [2.31, we have 

{-Ko{g)uQ,U(i)dg > 0. 



Combined with Theorem 4.7, we obtain 

I{(j),u,Ox)>0; 

{(j)(S)u,(j)(E)u)^= I I{(t),u,O)d[O]>0. 

We have thus proved that the Hermitian form (, )^ restricted to S(S>u is positive semidefinite, 
i.e., 

/ {Lo{MGi,MG2)m,mu,7T{g)u)d~g > 

JAIGi 

for every (f> & S. 

For an arbitrary iiT- finite vector / in llj{AIGi, MG2), write 

s 

/ = E /°(^) ® /fc (/" e a>(MOGi, M"G°), e u;(Af'Gi, M'G'2)). 

k=l 

For each fc, choose a sequence V'!"''' (2;) £ Gj?°(X") such that 

\^^^Hx)\<\f{x)\ 

Let V^^'^ = ELi V'i^^ ® f'k- Apparently, ip'^^^ G 5 and 

(c.(MGi, MG2)(5)^(^), ^(5)") ^ (c^(AfGi, MG2)(5)/, f)iu, n{~g)u) 
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pointwise. Furthermore, 

s 

\{uj{MGi,MG2){9)i^j,i^j){u,n{g)u)\ < ^ |(c.(MGi, MG2)(5)|A°|®/^, |/f 

k,i=l 

By the definition of semistable range, the function 

I ( (lu [MGi , MG^m I fk\(^fk,\f°\^n){u,7ri~g)u)\ 
is absolutely integrable on MGi (see Q). Hence, by dominated convergence theorem, 

(/ (g) u, / (g) u)^ = lim (g) u, ij}^^^ (g) w)^ > 

J-+00 

Therefore, the form (, is positive semidefinite. If (, does not vanish, then (, considered 
as a form on 

0(A/Gi,MG2)(7r) 

is positive definite ( see |^). We conclude that 6{MGi, MG2){t^) is unitary. Q.E.D. 

For (Gi,G2) in the stable range, the generic isotropic group Gi^ will be trivial. In this 
case, if tt is an irreducible unitary representation of il/Gi, then (, is positive semidefinite 
and nonvanishing. This result is due to Li ( ||Tl||). 

5.2 Gi = Sp2n(K) 

Take Gi = Sp27i{^) as an example. We can make our theorem more precise. First let me 
define a partial order < in M". We say that a ^ 6 if and only if 



k k 

3 



for all k. 



Corallary 5.1 Suppose n < p < q. Let tt be an irreducible unitary representation of 
MSp2n{'^)- Suppose for every leading exponent (see Chapter 8.8 in J^) v of tt we have 

^{v)-{^~n-\)^^p{Sp2nm)- 
Then (, )^ is positive semidefinite. In addition, if (, is nonvanishing, then 

e{MGi,MG2)i7T) 

is unitary. 

Proof: Take Vi — M^" and X2 — K"+^. Then V2 is a linear space equipped with a 
nondegenerate symmetric form of signature (p — n — 1,(7 — n — 1). We verify the conditions 
in Theorem 



5.1 



• For X e Horn {Vi,X2), the generic isotropic group Gi^, is just S'p„_i(]R) for n odd. 
For n even, the generic Gi^. can be identified with Spn~2{^) x N where N = R". One 
can easily check that S{g) for Sp2n(^) is integrable on G13;. 



23 



• Since 

ttq = uj{M'Gi, M'G^Y ® TT ® ^ has almost square integrable matrix coefficients. Ac- 
cording to Theorem 1 of [Q, ttq is weakly contained in L^(Gi). 

• The first condition is really redundant. By Theorem 3.2 |Tl|| , matrix coefficients 
of uj{MO{n + l,n + 1) , M Sp2n{'^)) are in L^-^ {MSp2yM)) for smaU 5 > 0. Since 
ttq is almost square integrable, the matrix coefficients of U!{MO{p, q), MSp2n{^)) fS" tt 
are in L^~^° {MG2) for small Sq > 0. Thus n must be in the semistable range of 
0{MGi,MG2). 

We conclude that (, is positive semidefinite. Q.E.D. 

5.3 G^ = 0{p,q) 
Similarly, we obtain 

Corallary 5.2 Suppose p + q < 2n + I. Let tt be an irreducible unitary representation of 
MO{p,q). Suppose for every leading exponent v of -k we have 

^[v)-{n-^^)<-p{0{p,q)). 

Then (, )^ is positive semidefinite. In addition, if (, is nonvanishing, then 

e{MGi,MG2)(Tr) 

is unitary. 

For p + q odd, the growth condition concerning the leading exponent v can be strengthened 
to allow 

^{v)-in-P±^)^-piOip,q)). 

The proof is omitted. 
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